Papers with token representations
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2020.tacl-1)
Copied to clipboard
| Challenge: | Pre-training methods like BERT mask individual words or subword units, but many tasks involve reasoning about relationships between two or more spans of text. |
| Approach: | They propose a pre-training method that masks contiguous random spans instead of random tokens to train the span boundary representations to predict the entire content of the masked span. |
| Outcome: | The proposed method outperforms BERT and its better-tuned baselines on span selection tasks and on coreference resolution tasks. |
Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale two-stream pre-trained models like CLIP have achieved tremendous success in image-text retrieval. |
| Approach: | They propose a cross-modal framework for image-text retrieval using two-stream pre-trained models . they embed images and texts into instance representations with two separate encoders . experimental results on MSCOCO and Flickr30k reveal the effectiveness of their framework . |
| Outcome: | The proposed framework improves image-text retrieval performance on two popular cross-modal retrieval benchmarks. |
High-order Refining for End-to-end Chinese Semantic Role Labeling (2020.aacl-main)
Copied to clipboard
| Challenge: | Current semantic role labeling methods are limited to short-term features and local decisions. |
| Approach: | They propose a high-order refining mechanism to perform interaction between all predicate-argument pairs via attention calculation. |
| Outcome: | The proposed model achieves state-of-the-art on Chinese SRL data, including CoNLL09 and Universal Proposition Bank, while relieving the long-range dependency issues. |
A Simple and Effective Dependency Parser for Telugu (2020.acl-srw)
Copied to clipboard
| Challenge: | Existing dependency parsers for Telugu use hand-crafted features based on linguistic information like part-of-speech and morphology which are expensive to annotate. |
| Approach: | They propose to replace linguistic feature templates with a minimal feature function for Telugu . they train a BERT model on the Telugus Wikipedia data and use contextual vector representations to train the parser. |
| Outcome: | The proposed parser achieves state-of-the-art for Telugu using contextual vector representations . the proposed model trains on the Telugus Wikipedia data and trains with a greedy transition based approach . |
Contrastive Learning for Many-to-many Multilingual Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing multilingual machine translation approaches focus on English-centric directions, while non-English directions lag behind. |
| Approach: | They propose a multilingual machine translation system with an emphasis on non-English directions. |
| Outcome: | The proposed model outperforms existing models on English-centric and non-English directions on multilingual translation benchmarks. |
Morpheme Matters: Morpheme-Based Subword Tokenization for Korean Language Models (2026.eacl-short)
Copied to clipboard
| Challenge: | Existing tokenizers rely on frequency-based segmentation to represent words . this often leads to inefficient token representations and oversegmentation . |
| Approach: | They propose a tokenization method that emphasizes the importance of Korean morphological structures in eojeol. |
| Outcome: | The proposed method outperforms existing tokenizers on Korean benchmark tasks and produces significantly fewer tokens per input sequence. |
How effective is BERT without word ordering? Implications for language understanding and data privacy (2021.acl-short)
Copied to clipboard
| Challenge: | Ordered word sequences contain the rich structures that define language. |
| Approach: | They show that token representations and self-attention activations within BERT are resilient to shuffling the order of input tokens. |
| Outcome: | The proposed model is able to handle shuffled token representations and self-attention activations . the model can handle GLUE language understanding tasks with bag-of-words data . |
Learning Source Phrase Representations for Neural Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to machine translation have been shown to be effective for long sentences . however, the attentional network can't capture long-distance dependencies . |
| Approach: | They propose a multi-head attention mechanism which generates phrase representations from token representations and incorporates them into the Transformer translation model to enhance its ability to capture long-distance relationships. |
| Outcome: | The proposed model can be computed in parallel and improves on the WMT 14 tasks. |
PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens. |
| Approach: | They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations. |
| Outcome: | Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods. |
Parameter-Efficient Tuning with Special Token Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | a recent study shows that parameter-efficient tuning is a challenge for multitask deployments. |
| Approach: | They propose a parameter-efficient tuning technique that only updates a small subset of parameters when adapting a pretrained model to downstream tasks. |
| Outcome: | The proposed method achieves comparable performance to fine-tuning in natural language understanding tasks including text classification and NER with only 0.029% of parameters trained. |
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on enhancing existing models with the primary objective of improving downstream performance on various NLP tasks. |
| Approach: | They propose to use BERT to encode meaningful knowledge in token representations to explain probing results. |
| Outcome: | The proposed model can detect syntactic and semantic abnormalities and distinguish between grammatical number and tense subspaces. |
MEMORY-VQ: Compression for Tractable Internet-Scale Memory (2024.naacl-short)
Copied to clipboard
Yury Zemlyanskiy, Michiel de Jong, Luke Vilnis, Santiago Ontanon, William Cohen, Sumit Sanghai, Joshua Ainslie
| Challenge: | Memory-based methods like LUMEN pre-compute token representations for retrieved passages to speed up inference. |
| Approach: | They propose a method to reduce storage requirements of memory-augmented models . they use a vector quantization variational autoencoder to compress token representations . |
| Outcome: | The proposed method achieves 16x compression rate with comparable performance on KILT benchmark. |
LangSAMP: Language-Script Aware Multilingual Pretraining (2025.acl-long)
Copied to clipboard
| Challenge: | Recent multilingual pretrained language models often avoid using language embeddings, which places a significant burden on token representations to encode all language-specific information. |
| Approach: | They propose a method that incorporates both language and script embeddings into the output of Transformer blocks before passing the final representations to the language modeling head for prediction. |
| Outcome: | The proposed method outperforms the baseline model in zero-shot crosslingual transfer across diverse downstream tasks. |
Long Document Summarization with Top-down and Bottom-up Inference (2023.findings-eacl)
Copied to clipboard
| Challenge: | Recent models infer latent representations of words or tokens with a transformer encoder, which is bottom-up and thus does not capture long-distance context well. |
| Approach: | They propose a method to infer latent representations of words or tokens in documents . they assume a hierarchical structure of a document where top-level captures long range dependency . |
| Outcome: | The proposed model can summarize an entire book and achieve competitive performance on a wide range of document summarization benchmarks. |
Can Edge Probing Tests Reveal Linguistic Knowledge in QA Models? (2022.coling-1)
Copied to clipboard
| Challenge: | grammatical knowledge is encoded in large pre-trained language models (LMs) this is done through supervised classification tasks to predict the grammamatical properties of a span using only the token representations coming from the LM encoder. |
| Approach: | They propose to use a supervised 'edge probing' task to detect grammatical knowledge in large pre-trained language models (LMs) this is done by encoding grammamatical properties using only token representations coming from the LM encoder. |
| Outcome: | The proposed model performs well when fine-tuned or in adversarial situations where the model is forced to learn wrong correlations. |
Take Off the Training Wheels! Progressive In-Context Learning for Effective Alignment (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have explored the working mechanisms of In-Context Learning (ICL) however, they mainly focus on classification and simple generation tasks, limiting their broader application to more complex generation tasks in practice. |
| Approach: | They propose an efficient Progressive In-Context Alignment method that embeds the task function learned from demonstrations into the separator token representation. |
| Outcome: | The proposed method surpasses vanilla ICL and achieves comparable performance to other alignment tuning methods. |
Better Pre-Training by Reducing Representation Confusion (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to improve pre-trained language models address information confusion in position encoding and model representations. |
| Approach: | They propose two techniques to improve pre-trained language models by decoupling directions and auxiliary regularizers. |
| Outcome: | The proposed techniques can improve pre-trained language models on GLUE benchmarks. |
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained MLMs produce an anisotropic distribution of token representations . this is not ideal for tasks that require discriminative semantic meanings of distinct tokens - a problem that exists in pre-training models . |
| Approach: | They propose a continual pre-training approach that encourages BERT to learn an isotropic distribution of token representations. |
| Outcome: | The proposed approach improves on a wide range of English and Chinese benchmarks. |
Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation (2025.emnlp-main)
Copied to clipboard
Qiyuan Chen, Hongsen Huang, Qian Shao, Jiahe Chen, Jintai Chen, Hongxia Xu, Renjie Hua, Ren Chuan, Jian Wu
| Challenge: | Large Language Models (LLMs) require high quality preference datasets to align with human preferences. |
| Approach: | They propose a framework that leverages inherent regulation of LLMs’ representation space for efficient and tailored preference dataset construction, named Icon2. |
| Outcome: | The proposed framework improves performance on benchmarks like AlpacaEval 2.0 and Arena-Hard while reducing computational costs by up to 48.1%. |
COPNER: Contrastive Learning with Prompt Guiding for Few-shot Named Entity Recognition (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for Named Entity Recognition (NER) use a similarity metric to measure semantic similarity between test samples and referents, but their performance is limited due to the label scarcity. |
| Approach: | They propose a novel approach to learn a similarity metric for measuring the semantic similarity between test samples and referents, where each referent represents an entity class. |
| Outcome: | The proposed approach outperforms state-of-the-art models with a significant margin in most cases. |
Manifold-Preserving Transformers are Effective for Short-Long Range Encoding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multi-head self-attention-based Transformers have shown promise in different learning tasks . but encoders of Transformers and their variants fail to preserve layer-wise contextual information . |
| Approach: | They propose an encoder model that guarantees a theoretical bound for layer-wise distance preservation between a pair of tokens. |
| Outcome: | The proposed model preserves equivalence between tokens and performs better than Transformers. |
GeoLAN: Geometric Learning of Latent Explanatory Directions in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models lack transparency and are often unable to explain causal relationships . |
| Approach: | They propose a training framework that treats token representations as geometric trajectories and applies stickiness conditions to the Kakeya Conjecture. |
| Outcome: | The proposed training framework maintains task accuracy while improving geometric metrics and reducing fairness biases. |
Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysis (2022.acl-long)
Copied to clipboard
| Challenge: | Structured sentiment analysis is a dependency parsing problem, with imbalanced label distributions and large text spans. |
| Approach: | They propose a novel labeling strategy which contains two sets of token pair labels . they propose tuple extraction problem with a more balanced label distribution . |
| Outcome: | The proposed model outperforms existing models on 5 benchmark datasets in four languages. |
SparseFlow: Accelerating Transformers by Sparsifying Information Flows (2024.acl-long)
Copied to clipboard
| Challenge: | SparseFlow is an efficient method to sparsify the dense information flows within transformers. |
| Approach: | They propose a method to sparsify the dense pathways of token representations across all transformer blocks by parameterizing them to be sparse. |
| Outcome: | The proposed method reduces computational costs by half on average without compromising task accuracy. |
Continual Few-shot Event Detection via Hierarchical Augmentation Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for continual few-shot event detection use labeled data, but in real-world applications, new event types emerge continually. |
| Approach: | They propose a memory-based framework for continual few-shot event detection . they incorporate prototypical augmentation into the memory set to memorize previous event types . |
| Outcome: | The proposed method outperforms existing methods in multiple continual few-shot event detection tasks. |
Scaling Within Document Coreference to Long Texts (2021.findings-acl)
Copied to clipboard
Raghuveer Thirukovalluru, Nicholas Monath, Kumar Shridhar, Manzil Zaheer, Mrinmaya Sachan, Andrew McCallum
| Challenge: | Existing end-to-end coreference resolution models use expensive span representations and antecedent prediction mechanisms. |
| Approach: | They propose an approximation to end-to-end coreference resolution models which scales gracefully to documents of any length. |
| Outcome: | The proposed model reduces training and inference time and memory costs compared to current models with minimal loss in accuracy. |
Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Language models rely on frequency information because they maximize the likelihood of tokens during training. |
| Approach: | They propose a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. |
| Outcome: | The proposed method reduces the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. |
SDR: Efficient Neural Re-ranking using Succinct Document Representation (2022.acl-long)
Copied to clipboard
| Challenge: | BERT based ranking models have been successful on various information retrieval tasks, but they are prone to storage and network fetching latency. |
| Approach: | They propose a late-interaction architecture that allows pre-computation of intermediate document representations, thus reducing latency. |
| Outcome: | The proposed model achieves 4x–11.6x higher compression rates on the MSMARCO passage re-reranking task compared to existing methods. |
Categorizing Semantic Representations for Neural Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Modern neural machine translation models suffer limitation in compositional generalization, resulting in weakened translation performance on unseen compounds. |
| Approach: | They propose to introduce categorization to the contextualized representations to improve generalization by reducing sparsity and overfitting. |
| Outcome: | The proposed method reduces compositional generalization error rates by 24% on a dedicated MT dataset. |
When Morphology Hides in Plain Sight: Breaking the Isolation in Vietnamese and Beyond (2026.acl-long)
Copied to clipboard
| Challenge: | Adaptive Boundary-Token Fusion and a Morpheme-Aware Attention Bias are used to encode monosyllabic morphemes. |
| Approach: | They propose a morpheme-aware Transformer that augments a pretrained Vietnamese encoder with two lightweight inductive biases. |
| Outcome: | The proposed morpheme-aware Transformer outperforms strong baselines on Vietnamese POS, NER, and sentence-level classification benchmarks. |
Holistic Sentence Embeddings for Better Out-of-Distribution Detection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have shown superior performance for textual OOD detection, but they estimate sample distance scores in the last-layer CLS embedding space. |
| Approach: | They propose to use token averaging and layer combination to boost OOD detection by deriving more holistic sentence embeddings. |
| Outcome: | The proposed method surpasses the state-of-the-art on a comprehensive suite of benchmarks by a 9.33% FAR95 margin. |
Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current language models process text as sequences of tokens that roughly correspond to words . individual tokens are often semantically unrelated to the meanings of the words/concepts they comprise . |
| Approach: | They propose a method to "read out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers. |
| Outcome: | The proposed method "reads out" the implicit vocabulary of an autoregressive LLM by examining differences in token representations across layers. |
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction (2023.acl-long)
Copied to clipboard
Jeremiah Milbauer, Annie Louis, Mohammad Javad Hosseini, Alex Fabrikant, Donald Metzler, Tal Schuster
| Challenge: | In many NLP tasks, the input text can be seen as a sequence of related segments. |
| Approach: | They propose a layer-adjustable interactions framework that contextualizes token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length. |
| Outcome: | The proposed model reduces 30-50% of attention FLOPs while maintaining high accuracy. |
Deep Span Representations for Named Entity Recognition (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing span-based NER systems shallowly aggregate token representations to span representations, causing performance degradations. |
| Approach: | They propose a method that shallowly aggregates token representations to span representations . DSpERT produces span representation of deep semantics with weight initialization . |
| Outcome: | The proposed model performs better than or competitive with state-of-the-art systems on six NER benchmarks. |
SLICER: Sliced Fine-Tuning for Low-Resource Cross-Lingual Transfer for Named Entity Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Large multilingual models fail to successfully transfer to low-resource languages for zero-shot cross-lingual transfer . sliced fine-tuning for named entity recognition (SLICER) forces stronger token contextualization in the Transformer. |
| Approach: | They propose a simple yet highly effective approach for improving zero-shot cross-lingual transfer for named entity recognition to low-resource languages. |
| Outcome: | The proposed approach reduces decontextualization of token representations and classifiers . it yields consistent transfer gains for low-resource languages, the authors show . |
A Meta-Learning Perspective on Transformers for Causal Language Modeling (2024.findings-acl)
Copied to clipboard
| Challenge: | Mechanisms of the Transformer architecture for causal language modeling are not well understood. |
| Approach: | They propose a meta-learning view of the Transformer architecture when trained for a causal language modeling task by explicating an inner optimization process that may happen within the Transformer. |
| Outcome: | The proposed model is based on a self-attention mechanism and has been widely used in natural language processing, computer vision, and scientific discovery. |
Information Flow Routes: Automatically Interpreting Language Models at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Current state-of-the-art language models (LMs) are built on top of the Transformer architecture. |
| Approach: | They propose to build graphs where nodes correspond to token representations and edges to computations . they show that attention heads and subword merging heads are important . |
| Outcome: | The proposed model can analyze behavior for specific types of predictions, or different domains. |
Multi-Attribute Steering of Language Models via Targeted Intervention (2025.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for steering large language models fail to scale to multi-attribute settings with conflicts, such as enhancing helpfulness while also reducing toxicity. |
| Approach: | They propose a steering framework for selective token-level intervention across multiple attributes that enforcing sparsity and orthogonality among vectors for different attributes. |
| Outcome: | The proposed framework outperforms existing ITI and parameter-efficient fine-tuning approaches across question answering tasks and generative tasks. |
SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models (2024.lrec-main)
Copied to clipboard
Zekun Wang, Jingchang Chen, Wangchunshu Zhou, Haichao Zhu, Jiafeng Liang, Liping Shan, Ming Liu, Dongliang Xu, Qing Yang, Bing Qin
| Challenge: | Experimental results show that SmartTrim accelerates the original model by 2-3 times with minimal performance degradation. |
| Approach: | They propose an adaptive acceleration framework which prunes redundant token representations and attention heads within each layer of the original model. |
| Outcome: | The proposed framework accelerates the original model by 2-3 times with minimal performance degradation across vision-language tasks. |
Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages (2025.emnlp-main)
Copied to clipboard
| Challenge: | Encoder transformer models compress information from all tokens into a single [CLS] token to represent global context. |
| Approach: | They propose a 1-D convolution module that augments token representations with multi-scale local features to improve performance. |
| Outcome: | Experiments on five diverse tasks show that the proposed framework outperforms baseline models by 1% to 14% while maintaining efficiency. |
Synergetic Interaction Network with Cross-task Attention for Joint Relational Triple Extraction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to joint entity-relation extraction are limited in their ability to capture the interdependence between the two sub-tasks. |
| Approach: | They propose a synergistic approach to capture interdependence between named entity recognition and relation extraction sub-tasks in a Synergetic Interaction Network. |
| Outcome: | The proposed model achieves significantly better performance on three benchmark datasets. |